Minimize bundled translations #1775

jotoeri · 2021-03-22T11:56:56Z

Why?

On Password-Policy the usage of SettingsSection instead of a simple div blew up the package-size from 100 KiB to 330 KiB of which 140 KB are the l10n.js of vue-Components. So i thought about how to reduce this.

Idea

The current implementation inserts the pot-file as json, and packs this within the bundle. However, there is a huge overhead on that kind of storing. So the idea was to reduce the overhead by storing only necessary data.
As model i initially used the json-format as translations are stored within apps.

Approach

Unfortunately i didn't find a npm-package/a way to store in the desired format, the recommended package (po2json) seems to be unmaintained and breaks on plural-translations.
Therefore this approach now manually transforms/compresses the format and decompresses it before inserting into gettextBuilder.
I must admit on the one hand this feels a bit hacky, as it compresses to a custom format on webpack-build, just to store the data and decompresses then on l10n insertion manually.
=> Webpack-build compresses -> Component stores compressed data -> l10n.js decompresses when loading gettext
The meta-information out of the header currently gets lost, but as far as i see, this works without. To be sure, on could think about restoring the plural-forms, language and content-type headers.

Result

This approach reduces the size of the bundled l10n.js from 142 KB to 32 KB!
Thus, the bundled password_policy settings reduce from 346 KiB to 237 KiB, while e.g. the forms-app bundle reduces from 2.75 MiB to 2.21 MiB.

More Information:

The way translations are stored currently (Only 1 plural and 2 singular translations):

{
  "locale": "de_DE",
  "json": {
    "charset": "utf-8",
    "headers": {
      "Last-Translator": "Mario Siegmann <[email protected]>, 2020",
      "Language-Team": "German (Germany) (https://www.transifex.com/nextcloud/teams/64236/de_DE/)",
      "Content-Type": "text/plain; charset=UTF-8",
      "Language": "de_DE",
      "Plural-Forms": "nplurals=2; plural=(n != 1);"
    },
    "translations": {
      "": {
        "": {
          "msgid": "",
          "comments": {
            "translator": "\nTranslators:\nPhilipp Fischbeck <[email protected]>, 2020\nProfDrJones <[email protected]>, 2020\nMark Ziegler <[email protected]>, 2020\nMario Siegmann <[email protected]>, 2020\n"
          },
          "msgstr": [
            "Last-Translator: Mario Siegmann <[email protected]>, 2020\nLanguage-Team: German (Germany) (https://www.transifex.com/nextcloud/teams/64236/de_DE/)\nContent-Type: text/plain; charset=UTF-8\nLanguage: de_DE\nPlural-Forms: nplurals=2; plural=(n != 1);\n"
          ]
        },
        "%n Action": {
          "msgid": "%n Action",
          "msgid_plural": "%n Actions",
          "msgstr": [
            "%n Aktion",
            "%n Aktionen"
          ]
        },
        "Actions": {
          "msgid": "Actions",
          "comments": {
            "reference": "src/components/Actions/Actions.vue:254"
          },
          "msgstr": [
            "Aktionen"
          ]
        },
        "Activities": {
          "msgid": "Activities",
          "comments": {
            "reference": "src/components/EmojiPicker/EmojiPicker.vue:176"
          },
          "msgstr": [
            "Aktivitäten"
          ]
        },
      }
    }
  }
}

The way i would store it here (same translation Data!):

{
  "locale": "de_DE",
  "translations": {
    "%n Action": {                       // Plural translation needs some specific handling
      "pluralId": "%n Actions",
      "msgstr": [
        "%n Aktion",
        "%n Aktionen"
      ]
    },
    "Actions": "Aktionen",           // Singular Translation
    "Activities": "Aktivitäten",
  }
},

Colorful image of the bundles before:

Colorful image of the bundles after:

raimund-schluessler · 2021-03-22T12:05:21Z

This approach reduces the size of the bundled l10n.js from 142 KB to 32 KB!
Thus, the bundled password_policy settings reduce from 346 KiB to 237 KiB, while e.g. the forms-app bundle forms reduces from 2.75 MiB to 2.21 MiB.

Why does the forms-app bundle size reduce more than the l10n.js file size? Is the same l10n.js file packed into the bundle multiple times?

jotoeri · 2021-03-22T12:09:02Z

Why does the forms-app bundle size reduce more than the l10n.js file size? Is the same l10n.js file packed into the bundle multiple times?

Yep, we currently import the components as single components (instead of nc/vue as a whole). And each component contains the l10n-file. You can see this also in the two bundle-analyzer-images.

raimund-schluessler · 2021-03-22T12:12:35Z

Why does the forms-app bundle size reduce more than the l10n.js file size? Is the same l10n.js file packed into the bundle multiple times?

Yep, we currently import the components as single components (instead of nc/vue as a whole). And each component contains the l10n-file. You can see this also in the two bundle-analyzer-images.

Hm, I thought (or maybe more like hoped) that this would be optimized by webpack.

skjnldsv · 2021-03-22T12:51:53Z

Hm, I thought (or maybe more like hoped) that this would be optimized by webpack.

I guess we have some improvements to do in our webpack config 🤔

webpack.common.js

skjnldsv

See comment

raimund-schluessler · 2021-03-22T22:14:17Z

Hm, I thought (or maybe more like hoped) that this would be optimized by webpack.

I guess we have some improvements to do in our webpack config 🤔

By the way, the font files also get duplicated in the final bundle. I had a look around, but couldn't find any solution for deduplicating this. If anyone has an idea, it would be great, since duplicating l10n.js and the fonts really has a huge impact on the bundle size.

@nextcloud/javascript

skjnldsv · 2021-03-23T07:21:04Z

Moving to webbpack5 might be a great thing to do and have better tree-shaking.

raimund-schluessler · 2021-03-23T07:40:02Z

Moving to webbpack5 might be a great thing to do and have better tree-shaking.

Not sure. In one of my apps I have webpack 5.27.1 and the code is still present multiple times in the final bundle. But I will check if using webpack5 in the vue-components as well has any influence.

jotoeri · 2021-03-23T08:51:42Z

As far as I see, webpack sees the components from the outside as one single block. So i think having separate chunks for l10n and fonts should enable webpack to see it as separate requirement and therefore enable to import it only once... I also tried that, but didn't manage to import it on a quick-shot. Seems to need some specific handling on the apps (or components?) to import the chunk then.

EDIT: Btw. also nice would be if we manage to import the chunk conditionally then. So as password_policy does not use the info-icon of settingsSection, the iconfonts package could be left out completely.... 🤔

jotoeri · 2021-03-23T10:12:42Z

⬆️ Small fixup to optimize plural storing a bit.

jotoeri · 2021-03-23T11:03:38Z

Opened a separate issue for deduplication, so we can keep track there. #1779

Signed-off-by: Jonas Rittershofer <[email protected]>

jotoeri · 2021-03-28T21:43:45Z

Somebody else up for a second review here? 😊

jotoeri added 3. to review Waiting for reviews discussion Need advices, opinions or ideas on this topic technical debt labels Mar 22, 2021

jotoeri requested review from PVince81, ChristophWurst, raimund-schluessler, juliusknorr, skjnldsv and marcoambrosini March 22, 2021 11:56

skjnldsv reviewed Mar 22, 2021

View reviewed changes

webpack.common.js Outdated Show resolved Hide resolved

skjnldsv requested changes Mar 22, 2021

View reviewed changes

skjnldsv added 2. developing Work in progress and removed 3. to review Waiting for reviews labels Mar 22, 2021

jotoeri force-pushed the enh/minimize_translations branch 2 times, most recently from 12984fc to 0f82fe5 Compare March 22, 2021 13:07

skjnldsv approved these changes Mar 22, 2021

View reviewed changes

skjnldsv added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Mar 22, 2021

jotoeri mentioned this pull request Mar 23, 2021

Deduplicate Bundles #1779

Closed

3 tasks

jotoeri removed the discussion Need advices, opinions or ideas on this topic label Mar 23, 2021

Compress translations on bundle

c1e09af

Signed-off-by: Jonas Rittershofer <[email protected]>

jotoeri force-pushed the enh/minimize_translations branch from 9b3f3da to c1e09af Compare March 24, 2021 13:54

skjnldsv requested review from julien-nc and korelstar March 24, 2021 14:27

ChristophWurst approved these changes Mar 29, 2021

View reviewed changes

ChristophWurst merged commit 6f27e09 into master Mar 29, 2021

ChristophWurst deleted the enh/minimize_translations branch March 29, 2021 08:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minimize bundled translations #1775

Minimize bundled translations #1775

jotoeri commented Mar 22, 2021 •

edited

Loading

raimund-schluessler commented Mar 22, 2021

jotoeri commented Mar 22, 2021

raimund-schluessler commented Mar 22, 2021

skjnldsv commented Mar 22, 2021

skjnldsv left a comment

raimund-schluessler commented Mar 22, 2021 •

edited

Loading

skjnldsv commented Mar 23, 2021

raimund-schluessler commented Mar 23, 2021

jotoeri commented Mar 23, 2021 •

edited

Loading

jotoeri commented Mar 23, 2021

jotoeri commented Mar 23, 2021

jotoeri commented Mar 28, 2021

Minimize bundled translations #1775

Minimize bundled translations #1775

Conversation

jotoeri commented Mar 22, 2021 • edited Loading

Why?

Idea

Approach

Result

More Information:

raimund-schluessler commented Mar 22, 2021

jotoeri commented Mar 22, 2021

raimund-schluessler commented Mar 22, 2021

skjnldsv commented Mar 22, 2021

skjnldsv left a comment

Choose a reason for hiding this comment

raimund-schluessler commented Mar 22, 2021 • edited Loading

skjnldsv commented Mar 23, 2021

raimund-schluessler commented Mar 23, 2021

jotoeri commented Mar 23, 2021 • edited Loading

jotoeri commented Mar 23, 2021

jotoeri commented Mar 23, 2021

jotoeri commented Mar 28, 2021

jotoeri commented Mar 22, 2021 •

edited

Loading

raimund-schluessler commented Mar 22, 2021 •

edited

Loading

jotoeri commented Mar 23, 2021 •

edited

Loading